2,712 research outputs found
A Reduction of the Elastic Net to Support Vector Machines with an Application to GPU Computing
The past years have witnessed many dedicated open-source projects that built
and maintain implementations of Support Vector Machines (SVM), parallelized for
GPU, multi-core CPUs and distributed systems. Up to this point, no comparable
effort has been made to parallelize the Elastic Net, despite its popularity in
many high impact applications, including genetics, neuroscience and systems
biology. The first contribution in this paper is of theoretical nature. We
establish a tight link between two seemingly different algorithms and prove
that Elastic Net regression can be reduced to SVM with squared hinge loss
classification. Our second contribution is to derive a practical algorithm
based on this reduction. The reduction enables us to utilize prior efforts in
speeding up and parallelizing SVMs to obtain a highly optimized and parallel
solver for the Elastic Net and Lasso. With a simple wrapper, consisting of only
11 lines of MATLAB code, we obtain an Elastic Net implementation that naturally
utilizes GPU and multi-core CPUs. We demonstrate on twelve real world data
sets, that our algorithm yields identical results as the popular (and highly
optimized) glmnet implementation but is one or several orders of magnitude
faster.Comment: 10 page
Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?
We prove that black-box variational inference (BBVI) with control variates,
particularly the sticking-the-landing (STL) estimator, converges at a geometric
(traditionally called "linear") rate under perfect variational family
specification. In particular, we prove a quadratic bound on the gradient
variance of the STL estimator, one which encompasses misspecified variational
families. Combined with previous works on the quadratic variance condition,
this directly implies convergence of BBVI with the use of projected stochastic
gradient descent. We also improve existing analysis on the regular closed-form
entropy gradient estimators, which enables comparison against the STL estimator
and provides explicit non-asymptotic complexity guarantees for both
Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference
Understanding the gradient variance of black-box variational inference (BBVI)
is a crucial step for establishing its convergence and developing algorithmic
improvements. However, existing studies have yet to show that the gradient
variance of BBVI satisfies the conditions used to study the convergence of
stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show
that BBVI satisfies a matching bound corresponding to the condition used
in the SGD literature when applied to smooth and quadratically-growing
log-likelihoods. Our results generalize to nonlinear covariance
parameterizations widely used in the practice of BBVI. Furthermore, we show
that the variance of the mean-field parameterization has provably superior
dimensional dependence.Comment: Accepted to ICML'23 for live oral presentatio
- …